Digital video creation, streaming, and playback are widely practiced activities. With increasing spatial resolutions, frame rates, and sample bit depths, and with high dynamic range content and 3D viewing options, users expect richer and more realistic experiences when playing back digital video on any device. Engineers use compression (also called source coding or source encoding) to reduce the bit rate of digital video. Compression decreases the cost of storing and transmitting video information by converting the information into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A “codec” is an encoder/decoder system.
Over the last 25 years, various video codec standards have been adopted, including the ITU-T H.261, H.262 (MPEG-2 or ISO/IEC 13818-2), H.263, H.264 (MPEG-4 AVC or ISO/IEC 14496-10) standards, the MPEG-1 (ISO/IEC 11172-2) and MPEG-4 Visual (ISO/IEC 14496-2) standards, and the SMPTE 421M (VC-1) standard. More recently, the H.265/HEVC standard (ITU-T H.265 or ISO/IEC 23008-2) has been approved. A video codec standard typically defines options for the syntax of an encoded video bitstream, detailing parameters in the bitstream when particular features are used in encoding and decoding. In many cases, a video codec standard also provides details about the decoding operations a video decoder should perform to achieve conforming results in decoding. Aside from codec standards, various proprietary codec formats define other options for the syntax of an encoded video bitstream and corresponding decoding operations.
Irrespective of which video codec standard or format is used, encoding video content is highly resource intensive. Whether video encoding is performed using general-purpose computer hardware (e.g., central processing units running software) or special-purpose computer hardware for “accelerated” encoding, video encoding still requires significant power and computational resources. These concerns are exacerbated on a low-power device such as a mobile phone, tablet computer, laptop computer, etc. On such a device, it is common to see a significant drop in battery life as soon as video encoding starts on the device, especially if a software implementation is used for the video encoding.
In a typical video sequence, most regions of a given frame are the same, or change only slightly, compared to frames before the given frame and after the given frame. Most video codec standards and formats use inter-picture prediction to exploit such frame-to-frame redundancy in a video sequence, and thereby improve quality or reduce bit rate of the video when compressed. For example, if a block of sample values in the given frame is predicted using inter-picture prediction, a video encoder estimates the motion of the block relative to one or more other, previously encoded/decoded frames, which are available for reference when encoding/decoding the given frame. The other, previously encoded/decoded frames are called reference frames. When the video encoder finds a matching block within a reference frame for the block of the given frame, the video encoder represents the matching block, e.g., using an identifier of the reference frame and the location of the matching block (relative to the location of the block of the given frame). The video encoder can determine differences between the block of the given frame and the matching block, then encode those differences. If the block of the given frame matches a block at the same location in a reference frame, the block of the given frame can be “skipped” by copying the “collocated” block from the reference frame. Similarly, if the entire given frame matches a reference frame, the given frame can be “skipped” by copying the reference frame. In either case (skipped block or skipped frame), the video encoder spends resources on encoding of the skipped block/frame, and the encoded video bitstream includes bits that represent the skipped block/frame.
Encoders for different video codec standards and formats have become very efficient at exploiting redundancy in digital video to improve quality and reduce bit rate. Such gains often have a high cost, however, in terms of usage computational resources and power.